MSU Video Quality Metrics Benchmark 2022
We help researchers test the performance of their metrics on the largest compression dataset
Key features of our benchmark
- Extensive and diverse dataset
- We used compression as distortion
- 40 different codecs of 10 compression standards
- 2500+ compressed streams with various target bitrates
- 780.000+ subjective responds
- VQA and IQA metrics
- 20+ metrics without variations
- 150+ metrics with variations
- Calculations over U and V planes
- Metrics with different weighted
average for planes
- Various charts
- Bar chart with overall quality
- Compare metrics performance on difference compression standards with 95% confidence intervals
- Illustrative Speed-Quality chart
Leaderboard
Chart below shows the correlation of metrics with subjective scores on our dataset.
Chart below shows more detailed results of tested metrics. For more information visit our site.
Speed-Quality Plot
Chart below shows Speed-Quality characteristics of metrics.
Methodology and Dataset
You can find detailed description here
How to submit
Send us an email to vqa@videoprocessing.ai with the file of .exe extension with your algorithm and instructions to run it from console. When launched from console we expect your file to have following options (or their analogs):
For Full-reference metrics:
- -
ref— path to reference video - -
dist— path to distorted video - -
output— path to output of your algorithm - -
t— threshold, if it's required in your algorithm
For No-reference metrics:
- -
dist— path to distorted video - -
output— path to output of your algorithm - -
t— threshold, if it's required in your algorithm
Feel free to add any addition information about your algorithm.
We won’t publish results of your algorithm without your permission.
If you have any suggestions or ideas how to improve our benchmark, please write us an e-mail on vqa@videoprocessing.ai.
Contacts
For questions and propositions, please contact us: vqa@videoprocessing.ai